1 link tagged with all of: etl + batch processing + infrastructure + airflow
Click any tag below to further narrow down your results
Links
A startup tech lead who’s only used cron seeks an orchestration tool for batch ETL jobs with dependency logic. They ask whether to pick Dagster or Airflow and question their relative resource usage. The community is invited to share experiences on reliability, scheduling flexibility, and system overhead.
- Airflow's scheduler/webserver can spike RAM, especially with hundreds of DAGs, and needs a metadata DB plus a message broker for Celery, adding operational overhead
- Dagster's code-first Python pipelines offer lighter UI, built-in lineage tracking, and lower resource use, particularly on Kubernetes or on-demand workers
- Airflow has a more mature ecosystem, larger community, and proven stability at scale, while Dagster's plugin/community support is less extensive but integrates well with dbt, Spark, and Snowflake
- The choice comes down to ecosystem maturity and infrastructure tolerance (Airflow) versus faster setup and operational simplicity (Dagster)